Macro and Micro Paradata for Survey Assessment

At present we typically assess quality by relying heavily on summary process measures or macro paradata that is a by-product of sample selection and survey administration (e.g., item and unit nonresponse rates, response variances, sampling and coverage errors). These measures are an outgrowth of the randomization-based approach to survey sampling that triumphed in government agencies around the world in the two decades after the seminal 1934 paper of Neyman. When most of our present measures were developed, therefore, nonsampling error problems, like nonresponse, were less serious or less well understood. Now there is a widespread belief that the randomization paradigm needs to be replaced by a more explicitly model-based approach that continues to incorporate features like random selection. In this paper we will illustrate how the current macro paradata measures might be revised or reinterpreted in the light of the concerns just mentioned. The changing partnership between data producers and data analyzers also increases the need for a greatly expanded use of micro paradata – i.e., process details known on each case (such as how many attempts it took to get an interview, whether the interview was in English or Spanish, etc.) To illustrative the paper will employ novel macro paradata summaries of coverage and unit nonresponse, plus micro paradata measures of response variation. The examples will all be taken from the National Survey of America’s Families (NSAF) but we believe the perspective we advocate would be actionable in other settings too.